Dynamic high-speed high-sensitivity imaging device and imaging method

ABSTRACT

Any one or both of an optical system with a structured lighting pattern and a structured detecting system having a plurality of regions with different optical characteristics are used. In addition, optical signals from an object to be observed through one or a small number of pixel detectors are detected while changing relative positions between the object to be observed and any one of the optical system and the detecting system, time series signal infoimation of the optical signals are obtained, and an image associated with an object to be observed from the time series signal information is reconstructed.

TECHNICAL FIELD

The present invention relates to dynamic high-speed high-sensitivityimaging technology in which an object to be observed and a detectingsystem having an optical system or a structure configured to projectstructured lighting are relatively displaced. Priority is claimed onJapanese Patent Application No. 2015-033520, filed Feb. 24, 2015, thecontent of which is incorporated herein by reference.

BACKGROUND ART

Japanese Unexamined Patent Application, First Publication No.2014-175819 (Patent Literature 1) discloses an imaging system includingelectromagnetic wave detecting elements arranged in a two-dimensionalarray. In an imaging device using array type detecting elements, thereare limitations on an imaging speed thereof from electrical restrictionswhen the elements are operated and a problem in that the imaging deviceis expensive and large in size.

Published Japanese Translation No. 2006-520893 of the PCT InternationalPublication (Patent Literature 2) discloses a device using a singlepixel detector. Furthermore, Japanese Patent No. 3444509 (PatentLiterature 3) discloses an image reading device having single pixeldetectors. An imaging device configured to perform single pixeldetection needs to spatiotemporally structure illumination light tocapture an image. For this reason, mechanical/electrical constraintsinvolved in spatiotemporally changing illumination light occur and thereare limitations on an imaging speed in the imaging device configured forsingle pixel detection.

For example, there are limitations on a speed of mechanically performingspatial-scanning with a laser in a confocal microscope and an imagecannot be captured at high speed. Ghost imaging is a method in whichnumerous different structural lightings are radiated using a spatiallight modulator or the like, detection is iterated, and an image isreconstructed. In such a method, since a speed of radiating lightingserves as a constraint, imaging is slow.

Japanese Unexamined Patent Application, First Publication No. 2013-15357(Patent Literature 4) discloses a flow cytometer using serialtime-encoded amplified microscopy (STEAM). In this publication, laserpulses with sufficiently wide wavelength widths are emitted from a laserirradiating unit at constant time intervals and the laser pulses aretwo-dimensionally dispersed by a two-dimensional spatial disperser.Different positions on a sample are irradiated with laser beams withwavelengths dispersed by the two-dimensional spatial disperser and thelaser beams are reflected. The reflected laser beams with thesewavelengths reversely pass through the two-dimensional spatial disperserso that the reflected laser beams return to one pulse.

Such a pulse passes through a Fourier transform, a frequency componentis converted into a time, and then the pulse is detected by aphotodiode. In a continuous time encoding amplitude microscope method,since a frequency (a wavelength) corresponds to a position on a sampleand a frequency component is converted into a time, the time hasinformation of the position on the sample. In other words, atwo-dimensional intensity distribution is converted into a time series.Information on surface structures of particles to be tested can beobtained from a temporal change in intensity signals of pulses acquiredin this way.

In a serial time-encoded amplified microscopy (STEAM), repetition onfrequency of a pulsed laser becomes constraints. Furthermore, an imagingdevice using STEAM is very expensive and large in size, an applicablelight wavelength range is limited to long wavelengths, and thus it isdifficult to achieve high sensitivity in a visible light range. For thisreason, there is a problem in that STEAM cannot be applied to a visiblefluorescence wavelength region necessary for application to the fieldsof life sciences/medicine.

CITATION LIST

-   [Patent Literature]-   [Patent Literature 1]

Japanese Unexamined Patent Application, First Publication No.2014-175819

-   [Patent Literature 2]

Published Japanese Translation No. 2006-520893 of the PCT InternationalPublication

-   [Patent Literature 3]

Japanese Patent No. 3444509

-   [Patent Literature 4]

Japanese Unexamined Patent Application, First Publication No. 2013-15357

SUMMARY OF INVENTION Technical Problem

Thus, the present invention is for the purpose of providing ahigh-speed, high-sensitivity, low-cost, and compact imaging device.

Solution to Problem

A first aspect of the present invention relates to a high-speed imagingmethod. The method includes using any one or both of an optical systemwith a structured lighting pattern and a structured detecting systemhaving a plurality of regions with different optical characteristics.Also, the method includes detecting optical signals from an object to beobserved through one or a small number of pixel detectors while changingrelative positions between the object to be observed and any one of theoptical system and the detecting system, obtaining time series signalinformation of the optical signals, and reconstructing an imageassociated with an object to be observed from the time series signalinformation.

A second aspect of the present invention relates to an imaging device. Afirst embodiment of the imaging device relates to an imaging devicehaving an optical system with a structured lighting pattern.

The imaging device has an optical system (11), one or a small number ofpixel detectors (15), a relative position control mechanism (17), and animage reconstructing unit (19).

The optical system (11) is an optical system with a structured lightingpattern having a plurality of regions with different opticalcharacteristics. The one or a small number of pixel detectors (15) is adetecting element configured to detect optical signals emitted by anobject to be observed (13) receiving light discharged from the opticalsystem (11).

The relative position control mechanism (17) is a mechanism configuredto change relative positions between the optical system (11) and theobject to be observed (13).

The image reconstructing unit (19) is an element configured toreconstruct an image of an object to be observed using optical signalsdetected by the one or a small number of pixel detectors (15).

The optical system (11) with a structured lighting pattern has aplurality of regions with different optical characteristics.

Examples of the optical signals include fluorescence, emitted light,transmitted light, or reflected light, but the present invention is notlimited thereto.

Examples of the optical characteristics include one or morecharacteristics (for example, transmission characteristics) of a lightintensity, a light wavelength, and polarization, but the presentinvention is not limited thereto.

Examples of the relative position control mechanism (17) include amechanism configured to change a position of the object to be observed(13) or a mechanism configured to change a position of the opticalsystem (11).

Examples of the image reconstructing unit (19) include an elementconfigured to reconstruct an image of an object to be observed usingoptical signals detected by one or a small number of pixel detectors(15) and information associated with a plurality of regions included inthe optical system (11) with the structured lighting pattern.

An imaging device of a second embodiment relates to one or a smallnumber of pixel detectors (55) having a plurality of regions withdifferent optical characteristics. The imaging device has an opticalsystem (51), one or a small number of pixel detectors (55), a relativeposition control mechanism (57), and an image reconstructing unit (59).

The optical system (51) is an element configured to irradiate an objectto be observed with light.

One or a small number of pixel detectors (55) are elements configured todetect optical signals emitted by the object to be observed (53)receiving light discharged from the optical system (51).

The relative position control mechanism (57) is a mechanism configuredto change relative positions between the optical system (51) and theobject to be observed (53) or relative positions between the object tobe observed (53) and the one or a small number of pixel detectors (55).

The image reconstructing unit (59) is an element configured toreconstruct an image of an object to be observed using optical signalsdetected by one or a small number of pixel detectors (55).

Examples of the relative position control mechanism (57) include amechanism configured to change a position of the object to be observed(53) or a mechanism configured to change a position of the one or asmall number of pixel detectors (55).

An example of the image reconstructing unit (59) is an elementconfigured to reconstruct an image of an object to be observed usingoptical signals detected by the one or a small number of pixel detectors(55) and information associated with a plurality of regions included inthe one or a small number of pixel detectors (57).

Advantageous Effects of Invention

According to the present invention, a high-speed imaging device whichcan fully utilize a band (a signal detection limit speed) of a single ornon-array type high-speed/high-sensitivity detectors in the world forthe first time (if a capacity is GHz or less, 10⁹ sheets (lines)/second)and greatly exceeds the speed limit of continuous imaging technology inthe related art can be provided.

According to the present invention, general-purpose and various types ofhigh-sensitivity imaging including visible fluorescence imaging whichwas impossible in imaging methods using a single pixel detector in therelated art can be performed using a universal optical system. Also,according to the present invention, since a simple optical system can beadopted, hardly any optical signal is lost and hardly any noise isintroduced. Thus, imaging with a high signal-to-noise (S/N) ratio can beperformed.

According to the present invention, since an optical system and anelectrical system which are used are simple, costs can be greatlydecreased and compactness can be achieved as compared with all imagingtechniques in the related art.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic constitution diagram showing that an object to beobserved moves in a first embodiment of an imaging device.

FIG. 2 is a schematic constitution diagram showing that a mechanismconfigured to change a position of an optical system (11) is provided inthe first embodiment of the imaging device.

FIG. 3 is a schematic constitution diagram showing that an object to beobserved moves in a second embodiment of an imaging device.

FIG. 4 is a schematic constitution diagram showing that the object to beobserved moves in the second embodiment of the imaging device.

FIG. 5 is a conceptual diagram showing that an object to be observedpasses through patterned lighting.

FIG. 6 is a conceptual diagram showing states of fluorescence emitted bythe object to be observed shown in FIG. 5.

FIG. 7 is a conceptual diagram showing a detection signal when thefluorescence emitted by the object to be observed shown in FIG. 5 hasbeen detected.

FIG. 8 is a conceptual diagram showing positions of fluorescencemolecules and fluorescence intensities obtained from detection signalintensities.

FIG. 9 is a view showing an image reproduction principle.

FIG. 10 is a view showing an example of an image reproducing process.

FIG. 11 is a flowchart showing an example of an image reconstructingprocess.

FIG. 12 is a view showing a matrix H.

FIG. 13 is a view showing a constitution of a target data vector f.

FIG. 14 is a schematic diagram showing an embodiment of an imagingdevice of the present invention.

FIG. 15 is a schematic diagram showing an embodiment of an imagingdevice of the present invention.

FIG. 16 is a schematic diagram of a device in Example 1.

FIG. 17 is a schematic constitution diagram showing a constitution inwhich an image is reproduced by detecting reflected light from an objectto be observed.

FIG. 18 is a schematic constitution diagram showing a constitution inwhich an image is reproduced by detecting fluorescence from an object tobe observed.

FIG. 19 is a view showing imaging when an overhead projector (OHP) sheetwith a black triangle printed thereon is used as an object to beobserved and the sheet is moved.

FIG. 20 shows detection results observed at time intervals. Going fromtop to bottom indicates elapse of time.

FIG. 21 is a graph showing change over time in a total amount of lightof optical signals obtained when an object to be observed has passedthrough patterned lighting.

FIG. 22 is an image of the object to be observed reconstructed from thegraph of FIG. 21.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a form configured to implement the present invention willbe described using the drawings. The present invention is not limited toa form which will be described below and also includes formsappropriately modified by a person of ordinary skill in the art in anobvious range from the following form. Note that radio signals,terahertz signals, radio frequency signals, acoustic signals, X-rays,y-rays, particle beams, or electromagnetic waves may be used in place ofoptical signals which will be described below. In this case, a lightsource which appropriately uses units configured to generate suchsignals and has a plurality of regions with different transmissioncharacteristics, reflection characteristics, or the like therefor may beappropriately used in place of the light source described below.Furthermore, as a structured lighting pattern or a structured detectingsystem, a pattern or a system obtained by using films in which asubstance changing transparency such as aluminum, silver, or lead ispartially applied or painted can be appropriately adopted.

FIG. 1 is a schematic constitution diagram showing that an object to beobserved moves in a first embodiment of an imaging device. The firstembodiment of the imaging device relates to an imaging device having anoptical system with a structured lighting pattern. The structuredlighting pattern means that there are a plurality of regions withdifferent light characteristics within a region of light with which theobject to be observed is irradiated.

As shown in FIG. 1, the imaging device has an optical system 11, one ora small number of pixel detectors 15, a relative position controlmechanism 17, and an image reconstructing unit 19.

The optical system 11 is an optical system (a system) including astructured lighting pattern having a plurality of regions with differentoptical characteristics. The optical system 11 may have a light source(not shown). In other words, examples of the optical system include agroup of optical elements having a light source (not shown) and filtersconfigured to receive light from the light source and form a structuredlighting pattern. Other examples of the optical system include a groupof light sources (or a group of optical elements including a group oflight source and optical elements) having a plurality of light sourcesconstituting a lighting pattern. Light from the light source passesthrough filters with, for example, an shown pattern of opticalcharacteristics and is radiated to an object to be measured to have anshown light pattern. The light source may be continuous light or pulsedlight, but is preferably continuous light. The light source may be whitelight or monochromatic light. Examples of the optical characteristicsinclude characteristics (for example, transmittance) associated with oneor more of a light intensity, a light wavelength, and polarization, butthe present invention is not limited thereto. Examples of a structuredlighting pattern having a plurality of regions with different opticalcharacteristics have a plurality of regions with a first light intensityand a plurality of regions with a second light intensity. Examples ofthe plurality of regions with different optical characteristics havesites with different light characteristics randomly distributed in acertain region. Furthermore, other examples of the plurality of regionswith different optical characteristics include a plurality of regionsdivided in a lattice shape, in which the plurality of regions have atleast regions having the first light intensity and regions having thesecond light intensity. The structured lighting pattern having theplurality of regions with different optical characteristics can beobtained, for example, by irradiating a transparent file with a patternprinted thereon with light from the light source, in addition to astructured lighting pattern described in examples. Light is radiated tothe object to be observed through the structured lighting pattern.

Examples of an object to be observed 13 can include various objects asan object to be observed depending on applications. Examples of theobject to be observed include cells, body fluids, and eyeballs (mayinclude moving eyeballs), but the present invention is not limitedthereto.

The one or a small number of pixel detectors 15 are detecting elementsconfigured to detect optical signals emitted by the object to beobserved 13 receiving light discharged from the optical system 11.Examples of the optical signals includes fluorescence, emitted light,transmitted light, or reflected light. Examples of one or a small numberof pixel detectors include a photomultiplier tube and a multi-channelplate photomultiplier tube, but the present invention is not limitedthereto. Since a small number of pixel detectors are compact and canoperate elements in parallel at high speed, a small number of pixeldetectors are preferably used for the present invention. Examples of asingle pixel detector are disclosed in Japan Patents Nos. 4679507 and

The relative position control mechanism 17 is a mechanism configured tochange relative positions between the optical system 11 and the objectto be observed 13. Examples of the relative position control mechanism17 include a mechanism configured to change a position of the object tobe observed 13 or a mechanism configured to change a position of theoptical system 11. Examples of the mechanism configured to change theposition of the object to be observed 13 include a mechanism having astage on which the object to be observed 13 can be mounted and anactuator configured to move the stage. Examples of the mechanismconfigured to change the position of the optical system 11 include amechanism configured to move a portion of the optical system 11 whichhas a plurality of regions and in which the structured lighting patternis formed (for example, only the light source, the filter and the lightsource) using an actuator or the like. The imaging device having themechanism configured to change the position of the object to be observed13 can be used for, for example, cell flow cytometry. Furthermore, sincethe size of the imaging device in this embodiment can be decreased, theimaging device in this embodiment can be used as an imaging device in awearable device having for example, a person's moving eyes as the objectto be observed. The imaging device having the mechanism configured tochange the position of the optical system 11 can be used as an imagingdevice in a microscope. Examples of such a microscope include a pointscanning type microscope, a confocal microscope, an electron microscope,a photo-acoustic microscope, and an ultrasonic microscope.

FIG. 2 is a schematic constitution diagram showing that the mechanismconfigured to change the position of the optical system 11 is providedin the first embodiment of the imaging device. In an example shown inFIG. 2, patterned lighting is moved so that places in the object to beobserved 13 are irradiated with light to have light characteristicsaccording to a pattern of the patterned lighting over time.

The image reconstructing unit 19 is a device configured to reconstructan image of the object to be observed using optical signals detected byone or a small number of pixel detectors 15. Examples of the imagereconstructing unit 19 include an image reconstructing unit configuredto reconstruct an image of the object to be observed using fluorescencedetected by one or a small number of pixel detectors 15 and informationassociated with the plurality of regions included in the optical system11 having the structured lighting pattern.

The image reconstructing unit 19 can be realized using, for example, acontrol device (for example, a computer) connected to the one or a smallnumber of pixel detectors 15. Such a control device is configured suchthat an input or output unit, a storage unit, a calculating unit, and acontrol unit are connected to each other through a bus or the like andthus information can be received or transmitted. Furthermore, thestorage unit stores various programs or numerical values such asparameters. When predetermined information is input from the input oroutput unit, such a control device reads a necessary program andnumerical values from the storage unit, and causes the calculating unitto perform predetermined calculation in accordance with the program, toappropriately store calculation results in the storage unit, and toperform an output from the input or output unit.

The image reconstructing unit 19, for example, has a time series signalinformation acquisition unit configured to receive optical signals for acertain period of time and acquire time series signal information of theoptical signals, a partial signal separating unit configured to separatepartial time series signal infon iation in a partial region of theobject to be observed from the time series signal information, a partialimage reconstructing unit configured to extract or reconstructinfoiination associated with images (emitted light intensities or thelike) of portions of the object to be observed from the acquired partialthe time series signal information of the object to be observed, and animage reconstructing unit configured to reconstruct the image associatedwith the object to be observed using the images of the portions of theobject to be observed which are reconstructed by the partial imagereconstructing unit.

Detection signals include information regarding a detected intensity forevery temporal change. The time series signal information acquisitionunit acquires the optical signals. Examples of the time series signalinformation acquisition unit include a time series signal informationacquisition unit configured to receive detection signals received,detected, and stored by the one or a small number of pixel detectors 15for a certain period of time as the time series signal information. Thetime series signal information acquired by the time series signalinformation acquisition unit may be appropriately stored in the storageunit. Furthermore, the time series signal information acquired by thetime series signal information acquisition unit is used for acalculating process using the partial signal separating unit. Thus, thetime series signal information may be transferred to the partial signalseparating unit.

The partial signal separating unit is an element configured to separatethe partial time series signal infolniation in the partial region of theobject to be observed from the time series signal information. The timeseries signal information includes detection signals derived from theportions of the object to be observed. For this reason, the partialsignal separating unit separates the partial time series signalinformation serving as the time series signal information in the partialregions of the object to be observed from the time series signalinformation. At this time, the partial signal separating unit readsstored information H associated with the lighting pattern and separatesthe partial time series signal information using the read information Hassociated with the lighting pattern and the time series signalinformation. In other words, the time series signal information includesvariation corresponding to the information H associated with thelighting pattern. Thus, the time series signal information can beseparated into the partial time series signal information using theinformation H associated with the lighting pattern. The partial timeseries signal information serving as the time series signal informationin the partial regions of the object to be observed may be appropriatelystored in the storage unit from the time series signal information.Furthermore, the partial time series signal information may betransferred to the partial image reconstructing unit for the purpose ofthe calculating process using the partial image reconstructing unit.

The partial image reconstructing unit is an element configured toextract or reconstruct information associated with the images (theemitted light intensities or the like) of the portions of the object tobe observed from the partial time series signal information. The partialtime series signal information is the time series signal information inthe partial regions. Thus, information f associated with the lightintensities in the regions can be obtained. The information associatedwith the images (the emitted light intensities or the like) of theportions of the object to be observed may be appropriately stored in thestorage unit. Furthermore, the information associated with the images(the emitted light intensities or the like) of the portions of theobject to be observed may be transferred to the image reconstructingunit for the purpose of the calculating process using the imagereconstructing unit. The image reconstructing unit is an elementconfigured to reconstruct the image associated with the object to beobserved using the images of the portions of the object to be observedreconstructed by the partial image reconstructing unit. The images ofthe portions of the object to be observed are images regions of theobject to be observed. Thus, the image associated with the object to beobserved can be reconstructed by matching the images.

An imaging device of a second embodiment relates to one or a smallnumber of pixel detectors 55 having a plurality of regions withdifferent light transmission performance. FIG. 3 is a schematicconstitution diagram showing that an object to be observed moves in thesecond embodiment of the imaging device. As shown in FIG. 3, the imagingdevice has an optical system 51, one or a small number of pixeldetectors 55, a relative position control mechanism 57, and an imagereconstructing unit 59. As long as a well-known optical system canirradiate the object to be observed with light, the well-known opticalsystem can be used as the optical system 51. The optical system 11 ofthe above-described first embodiment of the imaging device may be used.

The one or a small number of pixel detectors 55 are elements configuredto detect optical signals emitted by an object to be observed 53receiving light discharged from the optical system 51. The one or asmall number of pixel detectors 55 have sites having a plurality ofregions with different light transmission performance in addition to theone or a small number of pixel detectors 15 in the above-described firstembodiment of the imaging device. The plurality of regions withdifferent light transmission performance may be configured using, forexample, light filters present before a detecting unit. Such lightfilters have a plurality of regions with different light transmissionperformance. The plurality of regions may be divided, for example, in alattice shape, and the lattice may be divided such that lighttransparency is divided into two stages or more.

The relative position control mechanism 57 is a mechanism configured tochange relative positions between the optical system 51 and the objectto be observed 53 and relative positions between the object to beobserved 53 and the one or a small number of pixel detectors 55.Examples of the relative position control mechanism 57 is a mechanismconfigured to change a position of the object to be observed 53 or amechanism configured to change positions of the one or a smaller numberof pixel detectors 55. The mechanism configured to change a position ofthe object to be observed 53 can be used for, for example, cell flowcytometry, embedded micro-flow cytometry, and a wearable device. Theimaging device having the mechanism configured to change the positionsof the one or a small number of pixel detectors 55 can be used as, forexample, an imaging device mounted in a displaceable portion (forexample, a vehicle or wheels of a vehicle).

The image reconstructing unit 59 is an element configured to reconstructan image of the object to be observed using optical signals detected bythe one or a small number of pixel detectors 55. The imagereconstructing unit 19 in the above-described first embodiment of theimaging device may be used as the image reconstructing unit 59.

Examples of the image reconstructing unit 59 include an imagereconstructing unit configured to reconstruct the image of the object tobe observed using fluorescence detected by the one or a small number ofpixel detectors 55 and information associated with a plurality ofregions included in the one or a small number of pixel detectors 57.

Next, an example of an operation of the imaging device of the presentinvention will be described.

FIG. 5 is a conceptual diagram showing that an object to be observedpasses through patterned lighting. As shown in FIG. 5, an object to beobserved 13 is moved by a relative position control mechanism and thepatterned lighting passes through an optical system. The object to beobserved 13 has optical spatial infolniation, for example, fluorescencemolecules represented as F₁ to F₄. Furthermore, the fluorescencemolecules may not emit fluorescence depending on a received intensity oflight or have different intensities of emitted fluorescence. In otherwords, in this example, the fluorescence molecules represented as F₂first emit fluorescence and the emitted light intensity is affected bythe patterned lighting through which light passes. Light from the objectto be observed 13 may be appropriately focused through lenses or thelike. Furthermore, the light from the object to be observed 13 istransferred to the one or a small number of pixel detectors. In anexample of FIG. 5, a progressing direction of the object to be observedis set to an x axis and a y axis is provided in a directionperpendicular to the x axis which is on the same plane as the x axis. Inthis example, F₁ and F₂ which have the same y coordinates are observedas fluorescence on y₁ (which is denoted as H(x, y₁)). Furtheimore, F₃and F₄ which have the same y coordinates are observed as fluorescence ony₂ (which is denoted as H(x, r)).

FIG. 6 is a conceptual diagram showing states of the fluorescenceemitted by the object to be observed shown in FIG. 5. As shown in FIG.6, the fluorescence is caused to be emitted from the fluorescencemolecules, and for example, F₁ and F₂ experience the same lightingpattern. Thus, F₁ and F₂ are considered as having a similar timeresponse pattern or output pattern. On the other hand, it is conceivablethat F₁ and F₂ may have different emitted light intensities. For thisreason, the emitted light intensities of F₁ and F₂ can be approximatedto a product of F₁ and F₂ serving as coefficients specific to moleculesemitting light and H(x, y₁) serving as a time response pattern with thesame y₁ coordinates. The same applies to F₃ and F₄.

FIG. 7 is a conceptual diagram showing a detection signal when thefluorescence emitted by the object to be observed shown in FIG. 5 hasbeen detected. The detection signal is observed as sum signals of thefluorescence signals shown in FIG. 6. The signal include a temporalchange pattern of a plurality of intensities H(x, y_(n)). Coordinatesand fluorescence coefficients (fluorescence intensities) at thecoordinates can be obtained from a detection signal intensity (G(t)) andH(x, y_(n)).

FIG. 8 is a conceptual diagram showing positions of fluorescencemolecules and fluorescence intensities obtained from the detectionsignal intensities. As shown in FIG. 8, the fluorescence coefficients(the fluorescence intensities) F₁ to F₄ can be obtained from thedetection signal G(t).

The above-described principle will be described in greater detail. FIG.9 is a view showing an image reproduction principle. For example, it isassumed that there are F(1) and F(2) as in-target coordinates.Furthermore, at time 1, F(1) is irradiated with light of a first patternand F(2) is not irradiated with light of the first pattern. At time 2,F(1) is irradiated with light of a second pattern and F(2) is irradiatedwith light of the first pattern. At time 3, F(2) is not irradiated withlight and F(2) is irradiated with light of the second pattern. Thus, thedetection signal G(t) is as follows. G(1) =F(1)H(1), G(2)=F(1)H(2)F(2)H(1), and G(3)=F(2)H(2). Solving this, F(1) and

F(2) can be analyzed. If this principle is used, analysis is similarlyperformed even if the number of in-target coordinates is higher, andthus F(1) and F(n) can be obtained.

Subsequently when an object is in two dimensions, internal coordinatesof the object to be observed are set to F(x,y). On the other hand,patterned lighting is also set as having the same coordinates. If an xaxis direction is set to be n and a y axis direction is set to be n inthe internal coordinates of the object to be observed, the number ofunknown numbers of F(x,y) is n x n. Signals are measured as describedabove and an obtained signal G(t) is analyzed so that F(x,y) (0≤x≤n and0≤y≤n) can be reconstructed.

FIG. 10 is a view showing an example of an image reproducing process. Inthis example, an image is expressed by a determinant as f (an objectposition information vector). Furthermore, the patterned lighting isexpressed as H(X,y) and X is represented by a variable varying overtime. Detection signal intensities are expressed as g (a measurementsignal vector). Thus, the detection signal intensities can be expressedas g=Hf As shown in FIG. 10, both sides may be multiplied by an inversematrix H⁻¹ of H to obtain f. On the other hand, the inverse matrix H¹ ofH may not be easily obtained in some cases when H is too large. In thiscase, for example, a transposed matrix H^(t) of H may be used in placeof the inverse matrix. An initial estimated value f_(int) for f can beobtained using this relationship. After that, f is optimized using theinitial estimation value f_(int) for f so that the image of the objectto be observed can be reproduced.

In other words, FIG. 10 is a view showing an example of an imagereproducing process. In this example, an image is expressed by adeterminant as f (the object position information vector). Furthermore,the patterned lighting is expressed as H(X,y) and X is presented by avariable varying over time. Detection signal intensities are expressedas g (the measurement signal vector). Thus, the detection signalintensities can be expressed as g=Hf. As shown in FIG. 10, both sidesmay be multiplied by an inverse matrix H⁻¹ of H to obtain f. On theother hand, the inverse matrix H⁻¹ of H may not be easily obtained insome cases when H is too large. In this case, for example, the initialestimation value f_(init) for f can be obtained as results ofmultiplication between the transposed matrix H^(t) of H and g. Afterthat, f is optimized using the initial estimation value f_(init) for fso that the image of the object to be observed can be reproduced.

FIG. 11 is a flowchart showing an example of an image reconstructingprocess. FIG. 12 is a view showing a matrix H. FIG. 13 is a view showinga constitution of a target data vector f.

Imaging devices associated with another embodiment of the presentinvention can similarly reproduce an image of an object to be observedby applying the above-described principle.

FIG. 14 shows another embodiment of the imaging device of the presentinvention. The imaging device includes sites having a plurality ofregions with different light transmission performance on one or a smallnumber of pixel detectors in the imaging device of FIG. 1. The imagingdevice can distribute burdens on a lighting side and a sensor side. Forthis reason, characteristics which have not been observed in the relatedart among characteristics of an object such as observation of adistribution process can be observed.

FIG. 15 shows another embodiment of the imaging device of the presentinvention. The imaging device includes sites having a plurality ofregions with different light transmission performance on one or a smallnumber of pixel detectors in the imaging device of FIG. 2. The imagingdevice can distribute burdens on a lighting side and a sensor side. Forthis reason, for example, characteristics which have not been observedin the related art among characteristics of an object such asobservation of a distribution process can be observed.

Next, compressive sensing will be described.

Optical characteristics of a structured lighting pattern used by theimaging device are set to have different random distributions for everypixel so that the number of times of sampling is reduced and informationnecessary for reconstructing an image of an object to be observed isacquired. In other words, the number of times of sampling is reduced andthe image of the object to be observed is reconstructed on the basis ofscattered light obtained through a structured random lighting patternand sparsity of the object to be observed.

To be specific, the imaging device ascertains a range of a size of theobject to be observed by observing the object to be observed using therandomly distributed structured lighting pattern and performingreconstruction of the image a plurality of times. Subsequently, therange of the structured lighting pattern is reduced on the basis of therange of the size of the object to be measured to be able to cover arange necessary to reconstruct the image of the object to be observed.Alternatively, a region observed by the imaging device is expanded tomatch a region in which the object to be observed is present.

As described above, the imaging can improve throughput in image flowcytometry by designing the structured lighting pattern.

Note that the structured lighting pattern may be designed to have adelta function form in which autocorrelation between an opticalstructure and an optical structure itself becomes a state having a sharppeak. The autocorrelation of the optical structure is designed to havethe delta function form so that the structured lighting pattern anddetection signals when fluorescence emitted by the object to be observedhas been detected are uniquely determined. Thus, the image of the objectto be observed can be reconstructed.

Also, if the structured lighting pattern is designed to include manyregions through which light is not transmitted, overlapping of thedetection signals when the fluorescence emitted by the object to beobserved has been detected increases, and thus imaging with a highersimal-to-noise (SIN) ratio can be performed.

EXAMPLE 1

Next, the present invention will be described in detail using examples.

FIG. 16 is a schematic diagram of a device in Example 1. The devicerelates to a device in which the object to be observed moves, anirradiation pattern through which light is radiated to the object to beobserved is obtained using a light source and a mirror, and lighttransmitted through the object to be observed is observed so that animage of the object to be observed is reproduced.

An M4470L3-C1/blue (a wavelength of 47 nm) Olympus BX & Collimator forIX LED (1000 mA) manufactured by Thorlabs, Inc. was used as a lightsource. Note that, unlike a case in which coherent light such as a laseris used, in the case in which non-coherent light such as a lightemitting diode (LED) and a lamp was used, spots were not observed. Thus,accuracy was improved. In addition, a case in which continuous light wasused was more appropriate for high speed imaging than a case in whichpulsed light was used.

A silver mirror manufactured by Thorlabs, Inc. was used as the mirror.An optical axis of light incident on a spatial modulator was adjustedusing the silver mirror. A Digital Micromirror Device (DMD) DLPLCR 9000EVM manufactured by Texas Instruments was used. Light from the lightsource was structured in an optical system having a lighting patternthrough the spatial modulator. Note that, although a DMD was used inthis example, as long as a device can perform spatial modulation, lightmay be structured through a device other than a DMD. For example, anoverhead projector (OHP) sheet obtained by performing printing thereonand changing light transparency thereof in accordance with the printingmay be used and a transparent sheet with a microstructure may be used.Such lighting patterning is particularly preferably binary (light anddark) modulation. A spatial light modulator may be used to obtain thelighting pattern, but a problem such as zero order diffracted lightoccurs when a spatial light modulator is used.

Biconvex lenses manufactured by Thorlabs, Inc. were used as lenses. A 4fsystem was constructed using the lenses together with objective lensesand a structure on a spatial modulator was accurately opticallytransferred onto the object to be observed (a sample). A latter half (arear side of the object to be observed) of the 4f system is notessentially important and it is sufficient if the latter half thereofcan detect transmitted light from the sample with a good S/N.

An UPLSAPO20X manufactured by Olympus, Co. was used for the objectivelenses. The objective lenses have a function of forming an image ofstructured lighting serving as patterned lighting on the object to beobserved and a function of collecting optical signals from the object tobe observed. The objective lenses preferably have a high numericalaperture (NA) and a wide field of view to form many more images of thestructured lighting more finely.

An electric single axis stage HPS60-20x-SET and two rotary stagesKSPB-906M-M6 which were manufactured by SIGMAKOK1, Co., LTD. were usedas a sample stages for moving the object to be observed. An orientationof the object to be observed was three-dimensionally adjusted using thetwo rotary stages while the object to be observed was moved using thesingle axis stage.

An sCMOS camera Flash 4.0 manufactured by Hamamatsu Photonics K. K. wasused as a sensor. A pixel value of a captured image using such a camerawas integrated by a calculator and was set as a transmitted signal whichcould be obtained by a single pixel detector. Such a camera is for thepurpose of a principle demonstration test and preferably uses onehigh-speed pixel or a small number of high-speed detecting elements.

FIG. 17 is a schematic constitution diagram showing a constitution inwhich an image is reproduced by detecting reflected light from an objectto be observed. FIG. 18 is a schematic constitution diagram showing aconstitution in which an image is reproduced by detecting fluorescencefrom an object to be observed.

FIG. 19 is a view showing imaging when an OHP sheet with a blacktriangle printed thereon is used as an object to be observed and thesheet is moved. FIG. 20 shows detection results observed at timeintervals. Going from top to bottom indicates elapse of time. In atopmost detection result, a black triangle is present at a left part.Furthermore, a position of the black triangle moves in a rightwarddirection in observation results is portions below. From this, it can beseen that, if the object to be observed moves, a discharged signal canbe detected in accordance with displacement thereof Fitz. 21 is a graphshowing a temporal change of a total amount of light of optical signalsobtained when an object to be observed has passed through patternedlighting. FIG. 22 is an image of the object to be observed reconstructedfrom the graph of FIG. 21. From FIG. 22, it was shown that an image canbe reproduced so that the shape of the object to be observed can beascertained.

EXAMPLE 2

Next, multicolor imaging will be described. The multicolor imaging istechnology in which an object to be observed stained in multiple colorsusing a plurality of cell fluorescent labels is observed using acombination of a plurality of optical elements so that a color image isreconstructed. Note that the object to be observed is not limited tocells. Furthermore, light to be observed is not limited to fluorescence.A technique of dying an object to be observed is not limited to a cellfluorescent label and may use dyes or the like. The object to beobserved through the multicolor imaging is not limited to a stainedobject to be observed and may be a colored object to be observed.

Multi-color imaging of cells of which cell nuclei, cytoplasm, cellmembranes, or the like are stained in multiple colors, which has beenperformed in fluorescence activated cell sorting (FACS) in the relatedart, can be performed using a combination in which a plurality of cellfluorescent labels, dichroic mirrors, achromatic lenses, or band passfilters is further added to the above-described device shown in Example1. Note that emitted fluorescence light from multicolor-stained cellsmay be spectrally dispersed using optical elements such as diffractionelements instead of dichroic mirrors. In other words, various elementsusing refraction or diffraction can be used in spectroscopy for thepurpose of multi-color imaging.

To be specific, a device shown in Example 2 reconstructs an image ofcell membranes fluorescently stained red, an image of cytoplasmfluorescently stained green, and an image of cell nuclei fluorescentlystained blue. Subsequently, the device shown in Example 2 can generatean image of multicolor-stained cells by overlapping the reconstructedimages. The image of the multicolor-stained cells generated by thedevice shown in Example 2 is not inferior in comparison with an image ofmulticolor-stained cells captured using a camera capable of performingcolor imaging.

Note that, so far, as an example of the imaging device, although thedevice in which any one or both of the optical system with thestructured lighting pattern and the structured detecting system havingthe plurality of regions with different optical characteristics is used,the optical signals from the object to be observed is detected throughthe one or a small number of pixel detectors while changing the objectto be observed and the relative position of any one of theabove-described optical system and detecting system, the time seriessignal information of the optical signals is obtained, and the imageassociated with the object to be observed is reconstructed from the timeseries signal information has been described, the present invention isnot limited thereto. Tn other words, an imaging device is sufficient ifthe imaging device can acquire the above-described time series signalinformation of the optical signals and it is not essential toreconstruct the image associated with the object to be observed from thetime series signal information.

INDUSTRIAL APPLICABILITY

The present invention basically belongs to the field of optical devices,but can be used in various fields such as medical devices and wearabledevices.

REFERENCE SIGNS LIST

11 Optical system with structured lighting pattern

13 Object to be observed

15 One or small number of pixel detectors

17 Relative position control mechanism

19 Image reconstructing unit

51 Optical system

53 Object to be observed

55 One or a small number of pixel detectors

57 Relative position control mechanism

59 Image reconstructing unit

1-12. (canceled)
 13. A method for analyzing an object, comprising: (a)illuminating said object with light using at least one light source; (b)changing a position or an orientation of said object such that aplurality of regions of said object is sequentially illuminated by saidlight; (c) using at least one detector to sequentially detect aplurality of optical signals corresponding to said plurality of regionsof said object, wherein said plurality of optical signals comprise oneor more structured optical signals associated with a structured opticalpattern; and (d) obtaining time series information indicating a changein intensity of at least a subset of said plurality of optical signals.14. The method of claim 13, further comprising reconstructing one ormore time-independent images associated with said plurality of regionsof said object from said time series information.
 15. The method ofclaim 14, wherein said reconstructing of said one or moretime-independent images comprises: (a) separating partial time seriesinformation associated with a partial region of said plurality ofregions of said object from said time series information detected bysaid detector; (b) extracting information associated with one or morepartial images of said object from said partial time series information;and (c) reconstructing a whole image of said object using saidinformation associated with said one or more partial images of saidobject.
 16. The method of claim 13, wherein in (b), said position ofsaid object is changed (i) by moving a stage on which said object isdisposed or (ii) using cell flow cytometry.
 17. The method of claim 13,wherein said structured optical pattern is provided by at least oneoptical system selected from the group consisting of (i) an opticalelement having a light source and at least one filter or film configuredto receive light from said light source and form a structured lightingpattern, (ii) an optical system comprising a group of light sourcesconfigured to generate a structured lighting pattern, (iii) an opticalsystem comprising said light source and a spatial modulator to structurea light from said at least one light source to provide a lightingpattern, and (iv) an optical system comprising a light filter presentbefore said at least one detector.
 18. The method of claim 13, whereinsaid structured optical pattern is configured to have random intensitydistributions for every pixel.
 19. The method of claim 13, wherein saidstructured optical pattern is configured to have a delta function form.20. The method of claim 13, wherein said structured optical pattern isconfigured to include one or more regions through which light is nottransmitted.
 21. The method of claim 13, wherein said structured opticalpattern is disposed along a light path between said light source andsaid at least one detector, wherein said structured optical patterncomprises a plurality of regions with different optical characteristics.22. The method of claim 1, wherein said structured optical pattern isgenerated by: (a) ascertaining a range of size of said object byobserving said object using a randomly distributed structured lightingpattern and reconstructing an image associated with said object; and (b)adjusting said range of size of said structured lighting pattern basedon a size of said object or by changing a target region observed tomatch a reference region in which said object is present.
 23. A devicefor generating spatial information corresponding to time-independentimages of an object, comprising: (a) a light source configured toilluminate said object with light; (b) a relative position controlmechanism configured to change a position or an orientation of saidobject such that a plurality of regions of said object are sequentiallyilluminated by said light; and (c) at least one detector configured to(i) sequentially detect a plurality of optical signals corresponding tosaid plurality of regions of said object and (ii) obtain time seriesinformation indicating a change in intensity of at least a subset ofsaid optical signals, wherein said plurality of optical signalscomprises one or more structured optical signals associated with astructured optical pattern.
 24. The device of claim 23, furthercomprising an image reconstructor in communication with said detector,wherein said image reconstructor is configured to reconstruct one ormore time-independent images associated with said plurality of regionsof said object based on said time series information.
 25. The device ofclaim 24, wherein said image reconstructor comprises: (a) a partialsignal separating unit configured to separate partial time seriesinformation associated with a partial region of said plurality ofregions of said object from said time series information detected bysaid detector; (b) a partial image reconstructing unit configured toextract information associated with one or more partial images of saidobject from said partial time series information; and (c) an imagereconstructing unit configured to reconstruct a whole image of saidobject using said information associated with said one or more partialimages of said object.
 26. The device of claim 23, wherein said relativeposition control mechanism is configured to change said position of saidobject (i) by moving a stage on which said object is disposed or (ii)using flow cytometry.
 27. The device of claim 23, further comprising atleast one optical system configured to provide said structured opticalpattern, wherein said optical system is selected from the groupconsisting of (i) an optical elements having a light source and at leastone filter or film configured to receive light from the light source andform a structured lighting pattern, (ii) an optical system comprising agroup of light sources to constitute a structured lighting pattern,(iii) an optical system comprising a light source and a spatialmodulator to structure a light from said light source to have a lightingpattern, and (iv) an optical system comprising a light filter presentbefore said at least one detector.
 28. The device of claim 23, whereinsaid structured optical pattern is configured to have random intensitydistributions for every pixel.
 29. The device of claim 23, wherein saidstructured optical pattern is configured to have a delta function form.30. The device of claim 23, wherein said structured optical patterncomprises one or more regions through which light is not transmitted.31. The device of claim 28, wherein said structured optical pattern isgenerated by: (a) ascertaining a range of size of said object byobserving said object using a randomly distributed structured lightingpattern and reconstructing an image associated with said object; and (b)adjusting said range of size of said structured lighting pattern basedon a size of said object or by changing a target region observed tomatch a reference region in which said object is present.
 32. The deviceof claim 23, wherein said structured optical pattern is disposed along alight path between said light source and said detector, wherein saidstructured optical pattern comprises a plurality of regions withdifferent optical characteristics.